Expose TE fused MLP with module spec

See merge request ADLR/megatron-lm!3384

Co-authored-by: root <[email protected]>
Co-authored-by: William Dykas <[email protected]>

Moe inference functional tests

See merge request ADLR/megatron-lm!3403

…on H100

Co-authored-by: Oliver Koenig <[email protected]>

ci: Benchmark release tests suite with TE2.2 on H100

See merge request ADLR/megatron-lm!3458

Move data to GPU for TP data processing

See merge request ADLR/megatron-lm!3371

…, embedding tying"

This reverts commit 5ae21f8.

Signed-off-by: oliver könig <[email protected]>

…nd fix shape mismatch between vision and language transformer

Co-authored-by: Gao Deng <[email protected]>
Co-authored-by: Gao Deng <[email protected]>

Optimize dummy weight tensors for cudagraph and fix shape mismatch between vision and language transformer

See merge request ADLR/megatron-lm!3366

Add --enable-experimental to args.

See merge request ADLR/megatron-lm!3377

Co-authored-by: Zijie Yan <[email protected]>

perf(MLA): MLA down proj switch back to TELinear

See merge request ADLR/megatron-lm!3281

ci: Retry on network errors

See merge request ADLR/megatron-lm!3463

Co-authored-by: Oliver Koenig <[email protected]>
Co-authored-by: Guyue Huang <[email protected]>
Co-authored-by: Guyue Huang <[email protected]>

Add TE functional tests

See merge request ADLR/megatron-lm!3361

Signed-off-by: oliver könig <[email protected]>

Signed-off-by: oliver könig <[email protected]>

…to TELinear"

This reverts commit e63aee4.

Signed-off-by: oliver könig <[email protected]>

Signed-off-by: oliver könig <[email protected]>

…ns to CPU

Co-authored-by: Selvaraj Anandaraj <[email protected]>
Co-authored-by: Selvaraj Anandaraj <[email protected]>

Added support for offloading Swiglu activations to CPU

See merge request ADLR/megatron-lm!3024

Force inference to always gather logits with tensor parallelism

See merge request ADLR/megatron-lm!3442

…erate tokens

Only run prefill for requests that do not generate tokens

See merge request ADLR/megatron-lm!3499

Co-authored-by: Cyril Meurillon <[email protected]>
Co-authored-by: Cyril Meurillon <[email protected]>

Enable reruns by default

See merge request ADLR/megatron-lm!2739

…lidation feature

Co-authored-by: Ye Yu <[email protected]>
Co-authored-by: Chenhan Yu <[email protected]>

Clean up ModelOpt finetune scripts and add validation feature

See merge request ADLR/megatron-lm!3268

Fix typo in parallel_state expert parallelism

See merge request ADLR/megatron-lm!3548

… layers per stage in flexible pp layout

Fix cuda graph logic to determine first/layer layers per stage in flexible pp layout

See merge request ADLR/megatron-lm!3505

Remove extra barrier in checkpoint flow

See merge request ADLR/megatron-lm!3626

Fix error when TE is not installed

See merge request ADLR/megatron-lm!3625

…itializations and associated weight decay skipping.

Adding support for Spike No More embedding initializations and associated weight decay skipping.

See merge request ADLR/megatron-lm!3500

MiMo video VLM train example

See merge request ADLR/megatron-lm!3543

ci: Retry on `free(): invalid pointer`

See merge request ADLR/megatron-lm!3632

Signed-off-by: oliver könig <[email protected]>

Co-authored-by: Keshav Santhanam <[email protected]>
Co-authored-by: William Dykas <[email protected]>

Add Dynamic Backend Inference Tests

See merge request ADLR/megatron-lm!3475

…ate loading with PP>1 to ensure bit-wise match after saving and loading.

fix(distckpt, moe): Fix distckpt optimizer state loading with PP>1 to ensure bit-wise match after saving and loading.

See merge request ADLR/megatron-lm!3394

tests: Fix segfaults (maybe?)

See merge request ADLR/megatron-lm!3605

Co-authored-by: liu-zichen <[email protected]>

Fix mrope with context parallel

See merge request ADLR/megatron-lm!3603